MISC

# regarding 'readthedown' theme
# https://cran.r-project.org/web/packages/rmdformats/vignettes/introduction.html

Introduction

Check out this Kaggle

This data has been gathered at two solar power plants in India over a 34 day period. It has two pairs of files - each pair has one power generation dataset and one sensor readings dataset. The power generation datasets are gathered at the inverter level - each inverter has multiple lines of solar panels attached to it. The sensor data is gathered at a plant level - single array of sensors optimally placed at the plant.

There are a few areas of concern at the solar power plant -

Can we predict the power generation for next couple of days? - this allows for better grid management
Can we identify the need for panel cleaning/maintenance?
Can we identify faulty or sub-optimally performing equipment?

Data Dictionary for Power Generation data sets

AC_POWER : Amount of AC power generated by the inverter (source_key) in this 15 minute interval. Units - kW.
AC_ : Amount of DC power generated by the inverter (source_key) in this 15 minute interval. Units - kW.
DAILY_YIELD : Daily yield is a cumulative sum of power generated on that day, till that point in time.
DATE_TIME : Date and time for each observation. Observations recorded at 15 minute intervals.
PLANT_ID : Plant ID - this will be common for the entire file.
SOURCE_KEY : Source key in this file stands for the inverter id.
TOTAL_YIELD : This is the total yield for the inverter till that point in time.

Data Dictionary for Sensor Reading data sets

IRRADIATION: Amount of irradiation for the 15 minute interval.
DATE_TIME: Date and time for each observation. Observations recorded at 15 minute intervals.
PALNT_ID: Plant ID - this will be common for the entire file.
SOURCE_KEY: Stands for the sensor panel id. This will be common for the entire file because there’s only one sensor panel for the plant.
MODULE_TEMPERATURE: There’s a module (solar panel) attached to the sensor panel. This is the temperature reading for that module.
AMBIENT_TEMPERATURE: This is the ambient temperature at the plant.

Power Generation data set: Plant 1

Get and Split Data

p1.gd = read_csv('Plant_1_Generation_Data.csv') %>%
  slice_sample(prop = 0.10) %>% #!!<NOTE>temp, working with a sample of datset for speed purposes
  clean_names() %>%  #lowercase
  select(sort(tidyselect::peek_vars())) %>% #sort cols alphabetically
  select(where(is.factor),where(is.character),where(is.numeric)) #sort cols by data type

#OlsonNames()
#https://stackoverflow.com/questions/41479008/what-is-the-correct-tz-database-time-zone-for-india

p1.gd = p1.gd %>% mutate(
  date_time = as.POSIXct(strptime(p1.gd$date_time, "%d-%m-%Y %H:%M"), tz = 'Asia/Kolkata'),
  source_key = factor(p1.gd$source_key),
  source_key = factor(p1.gd$source_key)
) %>% rename(inverter = source_key)

p1.gd$plant_id = NULL

glimpse structure, and sample rows

p1.gd %>% slice_sample(n = 10) %>% DT::datatable()

p1.gd %>% glimpse()

## Rows: 6,877
## Columns: 6
## $ date_time   <dttm> 2020-05-27 14:15:00, 2020-06-03 05:15:00, 2020-05-16 1...
## $ inverter    <fct> WRmjgnKYAwPKWDb, 7JYdWkrLSPkdwr4, z9Y9gH1T5YWrNuG, YxYt...
## $ ac_power    <dbl> 442.12857, 0.00000, 217.78571, 0.00000, 0.00000, 0.0000...
## $ daily_yield <dbl> 4960.714, 0.000, 6312.857, 0.000, 5933.000, 0.000, 6975...
## $ dc_power    <dbl> 4502.8571, 0.0000, 2221.2857, 0.0000, 0.0000, 0.0000, 4...
## $ total_yield <dbl> 7122841, 7740873, 7020565, 7365310, 6400704, 7337918, 7...

check missing values

p1.gd %>% miss_var_summary()

## # A tibble: 6 x 3
##   variable    n_miss pct_miss
##   <chr>        <int>    <dbl>
## 1 date_time        0        0
## 2 inverter         0        0
## 3 ac_power         0        0
## 4 daily_yield      0        0
## 5 dc_power         0        0
## 6 total_yield      0        0

EDA: Factor Vars

counts each factor’s unique levels

sapply(p1.gd %>% select(where(is.factor)), n_unique) %>% as.data.frame()

##           .
## inverter 22

reference: names of unique levels

sapply(p1.gd %>% select(where(is.factor)), unique) %>% as.data.frame() %>% arrange()

##           inverter
## 1  WRmjgnKYAwPKWDb
## 2  7JYdWkrLSPkdwr4
## 3  z9Y9gH1T5YWrNuG
## 4  YxYtjZvoooNbGkE
## 5  adLQvlD726eNBSB
## 6  pkci93gMrogZuBj
## 7  iCRJl6heRkivqQ3
## 8  ih0vzX44oOqAx2f
## 9  1BY6WEcLGh8j5v7
## 10 bvBOhCH3iADSZry
## 11 ZoEaEvLYb1n2sOq
## 12 wCURE6d3bPkepu2
## 13 1IF53ai7Xc0U56Y
## 14 3PZuoBAID5Wc2HD
## 15 McdE0feGgRqW7Ca
## 16 rGa61gmuvPhdLxV
## 17 zBIq5rxdHJRwDNY
## 18 VHMLBKoKgIrUVDU
## 19 ZnxXDlPa8U1GXgE
## 20 zVJPv84UY57bAof
## 21 sjndEbLyjtCKgGv
## 22 uHbuxQJl8lW7ozc

viz: distribution of level counts

jpal = colorRampPalette(brewer.pal(8,'Dark2'))(22)

p1.gd %>% count(inverter) %>% plot_ly(y = ~fct_reorder(inverter,n), x = ~n, color = ~inverter, colors = jpal) %>% add_bars(hoverinfo = 'text', text = ~n) %>% hide_legend() %>% layout(
    title = 'Source Key Counts',
    xaxis = list(title = ''),
    yaxis = list(title = '')
    )

## Warning: `arrange_()` is deprecated as of dplyr 0.7.0.
## Please use `arrange()` instead.
## See vignette('programming') for more help
## This warning is displayed once every 8 hours.
## Call `lifecycle::last_warnings()` to see where this warning was generated.

Sanity Check - PASS - Pretty much a uniform distribution

EDA: Numeric Vars

viz bivariate numeric distribution

DataExplorer::plot_boxplot(p1.gd %>% select(where(is.numeric)), by = 'daily_yield')

viz: numeric univariate distributions

names.numeric = p1.gd %>% select(where(is.numeric)) %>% names

p1.gd %>% dlookr::plot_normality(
  names.numeric[1],
  names.numeric[2],
  names.numeric[3]
  )

The 0s for ac/dc power and daily_yield likely indicate the inverter was powered off for a ‘rest day’ / maintenance

viz: numeric univariate distributions

DataExplorer::plot_density(p1.gd %>% select(where(is.numeric)))

viz: distributions by ‘inverter’ factor

p1.gd %>% mutate(inverter = fct_reorder(.f = inverter, .x = daily_yield, .fun = median, .desc = TRUE)) %>%
  plot_ly(y = ~inverter, x = ~daily_yield, color = ~inverter, colors = jpal) %>% add_boxplot()%>%
  hide_legend() %>% layout(xaxis = list(title = ''), yaxis = list(title = ''), title = 'Distribution of Daily Yield by Inverter')

p1.gd %>% mutate(inverter = fct_reorder(.f = inverter, .x = ac_power, .fun = median, .desc = TRUE)) %>%
  plot_ly(y = ~inverter, x = ~ac_power, color = ~inverter, colors = jpal) %>% add_boxplot()%>%
  hide_legend() %>% layout(xaxis = list(title = ''), yaxis = list(title = ''), title = 'Distribution of AC Power by Inverter')

p1.gd %>% mutate(inverter = fct_reorder(.f = inverter, .x = dc_power, .fun = median, .desc = TRUE)) %>%
  plot_ly(y = ~inverter, x = ~dc_power, color = ~inverter, colors = jpal) %>% add_boxplot()%>%
  hide_legend() %>% layout(xaxis = list(title = ''), yaxis = list(title = ''), title = 'Distribution of DC Power by Inverter')

viz: ‘Rest Days’ Check

p1.gd %>% arrange(date_time) %>% group_nest(inverter) %>% unnest

## Warning: `cols` is now required when using unnest().
## Please use `cols = c(data)`

## # A tibble: 6,877 x 6
##    inverter        date_time           ac_power daily_yield dc_power total_yield
##    <fct>           <dttm>                 <dbl>       <dbl>    <dbl>       <dbl>
##  1 1BY6WEcLGh8j5v7 2020-05-15 00:15:00      0          0          0     6259559 
##  2 1BY6WEcLGh8j5v7 2020-05-15 05:00:00      0          0          0     6259559 
##  3 1BY6WEcLGh8j5v7 2020-05-15 05:30:00      0          0          0     6259559 
##  4 1BY6WEcLGh8j5v7 2020-05-15 08:15:00    385.       360.      3918.    6259919.
##  5 1BY6WEcLGh8j5v7 2020-05-15 10:15:00    650.      1308.      6637.    6260867.
##  6 1BY6WEcLGh8j5v7 2020-05-15 14:30:00    532.      4398.      5430.    6263957.
##  7 1BY6WEcLGh8j5v7 2020-05-15 16:00:00    447.      5275.      4551.    6264834.
##  8 1BY6WEcLGh8j5v7 2020-05-15 21:30:00      0       5754          0     6265313 
##  9 1BY6WEcLGh8j5v7 2020-05-16 05:45:00      0          0          0     6265313 
## 10 1BY6WEcLGh8j5v7 2020-05-16 06:15:00     29.9        3.88     309     6265317.
## # ... with 6,867 more rows

p1.gd.plots = p1.gd %>% arrange(date_time) %>% group_nest(inverter) %>% mutate(
  plots = map2(
    .x = data,
    .y = inverter,
    ~ggplot(data = .x, aes(date_time, daily_yield, color = daily_yield == 0, group = 1)) +
      geom_line(size = 1.2) + scale_color_manual(values = c('black','red')) + ggtitle(paste0('Inverter: ', .y))
  ))

p1.gd.plots$plots

## [[1]]

## 
## [[2]]

## 
## [[3]]

## 
## [[4]]

## 
## [[5]]

## 
## [[6]]

## 
## [[7]]

## 
## [[8]]

## 
## [[9]]

## 
## [[10]]

## 
## [[11]]

## 
## [[12]]

## 
## [[13]]

## 
## [[14]]

## 
## [[15]]

## 
## [[16]]

## 
## [[17]]

## 
## [[18]]

## 
## [[19]]

## 
## [[20]]

## 
## [[21]]

## 
## [[22]]

correlations: viz

p1.gd %>% dlookr::plot_correlate()

since dc and ac power are just perfectly convertible (like fahrenheit and celsius), they have a perfect correlation

EDA: Time Series Viz

Anomoly Plot

library(scales)
library(anomalize)
# anomalize(data, target, method = c("iqr", "gesd"), alpha = 0.05, max_anoms = 0.2, verbose = FALSE)

# alpha: Controls the width of the "normal" range. Lower values are more conservative while higher values are less prone to incorrectly classifying "normal" observations.
# max_anoms: The maximum percent of anomalies permitted to be identified.

p1.gd.anomalize = p1.gd %>% arrange(date_time) %>% 
  mutate(inverter = fct_reorder(inverter, -daily_yield)) %>% 
  group_by(inverter) %>%
  time_decompose(daily_yield, method = 'twitter', merge = TRUE) %>%
  anomalize(remainder, alpha = 0.05, method = 'gesd') %>%
  time_recompose()

ggplotly(
  p1.gd.anomalize %>%
    plot_anomalies(
      ncol = 2,
      alpha_dots = 0.5,
      alpha_circles = 0.5,
      size_circles = 2,
      time_recomposed = TRUE,
      alpha_ribbon = 0.05
      ) + scale_y_continuous(labels = comma) +
    labs(x = '', y = 'daily yield')
  ) %>% layout(showlegend = FALSE)

Power Generation data set: Plant 2

Get and Split Data

p2.gd = read_csv('Plant_2_Generation_Data.csv') %>%
  slice_sample(prop = 0.10) %>% #!!<NOTE>temp, working with a sample of datset for speed purposes
  clean_names() %>%  #lowercase
  select(sort(tidyselect::peek_vars())) %>% #sort cols alphabetically
  select(where(is.POSIXct), where(is.factor),where(is.character),where(is.numeric)) #sort cols by data type

#OlsonNames()
#https://stackoverflow.com/questions/41479008/what-is-the-correct-tz-database-time-zone-for-india

p2.gd = p2.gd %>% mutate(
  source_key = factor(p2.gd$source_key),
  source_key = factor(p2.gd$source_key)
) %>% rename(inverter = source_key)

p2.gd$plant_id = NULL

glimpse structure, and sample rows

p2.gd %>% slice_sample(n = 10) %>% DT::datatable()

p2.gd %>% glimpse()

## Rows: 6,769
## Columns: 6
## $ date_time   <dttm> 2020-05-15 17:15:00, 2020-06-10 18:00:00, 2020-05-17 2...
## $ inverter    <fct> NgDl19wMapZy17u, LYwnQax7tkwH5Cb, vOuJvMaM2sgwLmb, Mx2y...
## $ ac_power    <dbl> 195.793333, 6.740000, 0.000000, 0.000000, 2.285714, 213...
## $ daily_yield <dbl> 9410.466667, 2406.533333, 5358.000000, 8042.000000, 0.0...
## $ dc_power    <dbl> 199.726667, 7.000000, 0.000000, 0.000000, 2.371429, 217...
## $ total_yield <dbl> 111522001, 1795079359, 2231792, 2650151, 1795087577, 25...

check missing values

p2.gd %>% miss_var_summary()

## # A tibble: 6 x 3
##   variable    n_miss pct_miss
##   <chr>        <int>    <dbl>
## 1 date_time        0        0
## 2 inverter         0        0
## 3 ac_power         0        0
## 4 daily_yield      0        0
## 5 dc_power         0        0
## 6 total_yield      0        0

EDA: Factor Vars

counts each factor’s unique levels

sapply(p2.gd %>% select(where(is.factor)), n_unique) %>% as.data.frame()

##           .
## inverter 22

reference: names of unique levels

sapply(p2.gd %>% select(where(is.factor)), unique) %>% as.data.frame() %>% arrange()

##           inverter
## 1  NgDl19wMapZy17u
## 2  LYwnQax7tkwH5Cb
## 3  vOuJvMaM2sgwLmb
## 4  Mx2yZCDsyf6DPfv
## 5  Quc1TzYxW2pYoWX
## 6  4UPUqMRk7TRMgml
## 7  xoJJ8DcxJEcupym
## 8  Et9kgGMDl729KT4
## 9  Qf4GUc1pJu5T6c6
## 10 rrq4fwE8jgrTyWY
## 11 WcxssY2VbP4hApt
## 12 xMbIugepa2P7lBB
## 13 mqwcsP2rE7J0TFp
## 14 9kRcWv60rDACzjR
## 15 81aHJ1q11NBPMrL
## 16 LlT2YUhhzqhg5Sw
## 17 oZ35aAeoifZaQzV
## 18 PeE6FRyGXUgsRhN
## 19 q49J1IKaHRwDQnt
## 20 oZZkBaNadn6DNKz
## 21 V94E5Ben1TlhnDV
## 22 IQ2d7wF4YD8zU1Q

viz: distribution of level counts

jpal = colorRampPalette(brewer.pal(8,'Dark2'))(22)

p2.gd %>% count(inverter) %>% plot_ly(y = ~fct_reorder(inverter,n), x = ~n, color = ~inverter, colors = jpal) %>% add_bars(hoverinfo = 'text', text = ~n) %>% hide_legend() %>% layout(
    title = 'Source Key Counts',
    xaxis = list(title = ''),
    yaxis = list(title = '')
    )

Sanity Check - PASS - Pretty much a uniform distribution

EDA: Numeric Vars

viz bivariate numeric distribution

DataExplorer::plot_boxplot(p2.gd %>% select(where(is.numeric)), by = 'daily_yield')

DataExplorer::plot_boxplot(p2.gd %>% select(where(is.numeric)), by = 'total_yield')

viz: numeric univariate distributions

names.numeric = p2.gd %>% select(where(is.numeric)) %>% names

p2.gd %>% dlookr::plot_normality(
  names.numeric[1],
  names.numeric[2],
  names.numeric[3],
  names.numeric[4]
  )

Seems to be many outliers, especially 0s, in several features. Indicative of faulty inverters? Plot by inverter

viz: numeric univariate distributions

DataExplorer::plot_density(p2.gd %>% select(where(is.numeric)))

viz: distributions by ‘inverter’ factor

p2.gd %>% mutate(inverter = fct_reorder(.f = inverter, .x = total_yield, .fun = median, .desc = TRUE)) %>%
  plot_ly(y = ~inverter, x = ~total_yield, color = ~inverter, colors = jpal) %>% add_boxplot()%>%
  hide_legend() %>% layout(xaxis = list(title = ''), yaxis = list(title = ''), title = 'Distribution of Total Yield by Inverter')

p2.gd %>% mutate(inverter = fct_reorder(.f = inverter, .x = daily_yield, .fun = median, .desc = TRUE)) %>%
  plot_ly(y = ~inverter, x = ~daily_yield, color = ~inverter, colors = jpal) %>% add_boxplot()%>%
  hide_legend() %>% layout(xaxis = list(title = ''), yaxis = list(title = ''), title = 'Distribution of Daily Yield by Inverter')

p2.gd %>% mutate(inverter = fct_reorder(.f = inverter, .x = ac_power, .fun = median, .desc = TRUE)) %>%
  plot_ly(y = ~inverter, x = ~ac_power, color = ~inverter, colors = jpal) %>% add_boxplot()%>%
  hide_legend() %>% layout(xaxis = list(title = ''), yaxis = list(title = ''), title = 'Distribution of AC Power by Inverter')

p2.gd %>% mutate(inverter = fct_reorder(.f = inverter, .x = dc_power, .fun = median, .desc = TRUE)) %>%
  plot_ly(y = ~inverter, x = ~dc_power, color = ~inverter, colors = jpal) %>% add_boxplot()%>%
  hide_legend() %>% layout(xaxis = list(title = ''), yaxis = list(title = ''), title = 'Distribution of DC Power by Inverter')

viz: ‘Rest Days’ Check

p2.gd %>% arrange(date_time) %>% group_nest(inverter) %>% unnest

## Warning: `cols` is now required when using unnest().
## Please use `cols = c(data)`

## # A tibble: 6,769 x 6
##    inverter        date_time           ac_power daily_yield dc_power total_yield
##    <fct>           <dttm>                 <dbl>       <dbl>    <dbl>       <dbl>
##  1 4UPUqMRk7TRMgml 2020-05-15 04:45:00      0          0         0      2429011 
##  2 4UPUqMRk7TRMgml 2020-05-15 06:15:00     26.5        6.13     27.4    2429017.
##  3 4UPUqMRk7TRMgml 2020-05-15 06:45:00    154.        39.5     158.     2429050.
##  4 4UPUqMRk7TRMgml 2020-05-15 10:15:00      0       2171         0      2431182 
##  5 4UPUqMRk7TRMgml 2020-05-15 10:45:00      0       2171         0      2431182 
##  6 4UPUqMRk7TRMgml 2020-05-15 13:00:00   1229.      2588.     1261.     2431599.
##  7 4UPUqMRk7TRMgml 2020-05-15 13:45:00      0       2750         0      2431761 
##  8 4UPUqMRk7TRMgml 2020-05-15 15:45:00    749.      3420.      765.     2432431.
##  9 4UPUqMRk7TRMgml 2020-05-15 17:15:00    194.      4132.      198.     2433143.
## 10 4UPUqMRk7TRMgml 2020-05-15 19:15:00      0       4201         0      2433212 
## # ... with 6,759 more rows

p2.gd.plots = p2.gd %>% arrange(date_time) %>% group_nest(inverter) %>% mutate(
  plots = map2(
    .x = data,
    .y = inverter,
    ~ggplot(data = .x, aes(date_time, daily_yield, color = daily_yield == 0, group = 1)) +
      geom_line(size = 1.2) + scale_color_manual(values = c('black','red')) + ggtitle(paste0('Inverter: ', .y))
  ))

p2.gd.plots$plots

## [[1]]

## 
## [[2]]

## 
## [[3]]

## 
## [[4]]

## 
## [[5]]

## 
## [[6]]

## 
## [[7]]

## 
## [[8]]

## 
## [[9]]

## 
## [[10]]

## 
## [[11]]

## 
## [[12]]

## 
## [[13]]

## 
## [[14]]

## 
## [[15]]

## 
## [[16]]

## 
## [[17]]

## 
## [[18]]

## 
## [[19]]

## 
## [[20]]

## 
## [[21]]

## 
## [[22]]

correlations: viz

p2.gd %>% dlookr::plot_correlate()

since dc and ac power are just perfectly convertible (like fahrenheit and celsius), they have a perfect correlation

EDA: Time Series Viz

Anomoly Plot

library(scales)
library(anomalize)
# anomalize(data, target, method = c("iqr", "gesd"), alpha = 0.05, max_anoms = 0.2, verbose = FALSE)

# alpha: Controls the width of the "normal" range. Lower values are more conservative while higher values are less prone to incorrectly classifying "normal" observations.
# max_anoms: The maximum percent of anomalies permitted to be identified.

p2.gd.anomalize = p2.gd %>% arrange(date_time) %>% 
  mutate(inverter = fct_reorder(inverter, -daily_yield)) %>% 
  group_by(inverter) %>%
  time_decompose(daily_yield, method = 'twitter', merge = TRUE) %>%
  anomalize(remainder, alpha = 0.05, method = 'gesd') %>%
  time_recompose()

ggplotly(
  p2.gd.anomalize %>%
    plot_anomalies(
      ncol = 2,
      alpha_dots = 0.5,
      alpha_circles = 0.5,
      size_circles = 2,
      time_recomposed = TRUE,
      alpha_ribbon = 0.05
      ) + scale_y_continuous(labels = comma) +
    labs(x = '', y = 'daily yield')
  ) %>% layout(showlegend = FALSE)

Notes

better Total Yield viz to show incremental change – maybe diff?

Power Plants

Power Plants

MISC

Introduction

Data Dictionary for Power Generation data sets

Data Dictionary for Sensor Reading data sets

Power Generation data set: Plant 1

Get and Split Data

glimpse structure, and sample rows

check missing values

EDA: Factor Vars

counts each factor’s unique levels

reference: names of unique levels

viz: distribution of level counts

EDA: Numeric Vars

viz bivariate numeric distribution

viz: numeric univariate distributions

viz: numeric univariate distributions

viz: distributions by ‘inverter’ factor

viz: ‘Rest Days’ Check

correlations: viz

EDA: Time Series Viz

Anomoly Plot

Power Generation data set: Plant 2

Get and Split Data

glimpse structure, and sample rows

check missing values

EDA: Factor Vars

counts each factor’s unique levels

reference: names of unique levels

viz: distribution of level counts

EDA: Numeric Vars

viz bivariate numeric distribution

viz: numeric univariate distributions

viz: numeric univariate distributions

viz: distributions by ‘inverter’ factor

viz: ‘Rest Days’ Check

correlations: viz

EDA: Time Series Viz

Anomoly Plot

Notes